Synthesis of visible speech

Given the importance of visible information in face-toface communication, visible speech synthesis is being developed to control and manipulate visible speech. Experiments have shown that this visible speech is particularly important when the auditory speech is degraded, because of noise, bane:width filtering, or hearing impairment (Massaro, 1987). The strong influence of visible speech is not limited to situations with degraded auditory input, however; it occurs even when visible speech is paired with perfectly intelligible speech sounds. The influence of visible speech is easily experienced in a demonstration of a McGurk effect (McGurk & MacDonald, 1976). A videotape of a person making a visible labial articulation Ipa-pal is dubbed with the alveolar nasal speech sounds Ina-na/. This dubbed speech event gives a situation in which intelligible auditory speech is paired with a contradictory visual articulation. A strong effect of the visual source of information is observed, with the viewer often reporting hearing the labial nasal lma-mal. It can be shown that the viewer's experience is influenced by both the audible and the visible speech. The value of visible speech in speech perception warrants that visible speech should be studied, just as auditory speech has been studied. Although some progress has been made using natural speech as stimuli, the synthesis of a speaker's face permits a better controlled and more systematic analysis of the perceptual process. Given that synthetic auditory speech has proven valuable for the study of auditory speech perception, visible speech synthesis should be a valuable tool for the study of visual and bimodal (auditory-visual) speech perception. In addition to enabling exact control over the speech stimulus, synthetic speech allows the creation of novel speech segments that are not easy to produce naturally. Presenting novel stimuli to subjects in psychophysical tasks provides important information about the processes involved in perception.